We revisit the Adaptive Stochastic Quantization (ASQ) problem and present algorithms that find optimal solutions with asymptotically improved time and space complexities.

We revisit the Adaptive Stochastic Quantization (ASQ) problem and present algorithms that find optimal solutions with asymptotically improved time and space complexities.

Such compression problems naturally arise in distributed and federated learning.

Recent work has shown that 8-bit floating point (FP8) can be used for efficiently training neural networks with reduced computational overhead compared to training in FP32/FP16. In this work, we investigate the use of FP8 training in a federated learning context. This brings not only the usual benefits of FP8 which are desirable for on-device training at the edge, but also reduces client-server communication costs due to significant weight compression. We present a novel method for combining FP8 client training while maintaining a global FP32 server model and provide convergence analysis. Experiments with various machine learning models and datasets show that our method consistently yields communication reductions of at least 2.9x across a variety of tasks and models compared to an FP32 baseline.

This paper introduces an approach to employ clipped uniform quantization in federated learning settings, aiming to enhance model efficiency by reducing communication overhead without compromising accuracy. We explore the implications of symmetric clipping and uniform quantization on model performance, highlighting the utility of stochastic quantization to mitigate quantization artifacts and improve model robustness. Through extensive simulations on the MNIST dataset, our results demonstrate that the proposed method achieves near full-precision performance while ensuring substantial communication savings. Specifically, our approach facilitates efficient weight averaging based on quantization errors, effectively balancing the trade-off between communication efficiency and model accuracy. The comparative analysis with conventional quantization methods further confirms the superiority of our technique.

Needless to be exemplified by Boltzmann machine, Amari-Hopfield model and diffusion models, fundamental physics has provided a great influence on machine learning. Then a natural question arises -- to what extent do the fundamental physics and machine learning overlap with each other? For example, the notion of quantum is the central concept in microscopic physics. To what extent can quantum mechanics be formulated in terms of neural networks? Partial answers to this interesting question come from two developments at the intersection of machine learning and physics: (1) Gaussian processes and (2) stochastic neurodynamics, which we shall describe in order. Both of these have their roots in the research of random neural networks initiated by Amari [1, 3] and Rozonoer [2]. The random neural network is a fundamental tool to reveal the macroscopic properties of typical neural networks, as well as a key to control complicated learning dynamics of neural networks.

Most real-world applications that employ deep neural networks (DNNs) quantize them to low precision to reduce the compute needs. We present a method to improve the robustness of quantized DNNs to white-box adversarial attacks. We first tackle the limitation of deterministic quantization to fixed ``bins'' by introducing a differentiable Stochastic Quantizer (SQ). We explore the hypothesis that different quantizations may collectively be more robust than each quantized DNN. We formulate a training objective to encourage different quantized DNNs to learn different representations of the input image. The training objective captures diversity and accuracy via mutual information between ensemble members. Through experimentation, we demonstrate substantial improvement in robustness against $L_\infty$ attacks even if the attacker is allowed to backpropagate through SQ (e.g., > 50\% accuracy to PGD(5/255) on CIFAR10 without adversarial training), compared to vanilla DNNs as well as existing ensembles of quantized DNNs. We extend the method to detect attacks and generate robustness profiles in the adversarial information plane (AIP), towards a unified analysis of different threat models by correlating the MI and accuracy.

In this work, we establish a direct connection between generative diffusion models (DMs) and stochastic quantization (SQ). The DM is realized by approximating the reversal of a stochastic process dictated by the Langevin equation, generating samples from a prior distribution to effectively mimic the target distribution. Using numerical simulations, we demonstrate that the DM can serve as a global sampler for generating quantum lattice field configurations in two-dimensional $\phi^4$ theory. We demonstrate that DMs can notably reduce autocorrelation times in the Markov chain, especially in the critical region where standard Markov Chain Monte-Carlo (MCMC) algorithms experience critical slowing down. The findings can potentially inspire further advancements in lattice field theory simulations, in particular in cases where it is expensive to generate large ensembles.